Skip to content

feat(idempotency): add opt-in managed request idempotency - #47

Merged
ALikesToCode merged 3 commits into
feat/gateway-alertsfrom
feat/managed-idempotency
Oct 9, 2026
Merged

ALikesToCode merged 3 commits into
feat/gateway-alertsfrom
feat/managed-idempotency

Conversation

@ALikesToCode

@ALikesToCode ALikesToCode commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds opt-in idempotency for managed non-streaming chat behind MANAGED_IDEMPOTENCY_ENABLED, which defaults to false.

With the flag off:

  • The intelligence gateway keeps rejecting Idempotency-Key with idempotency_not_supported.
  • Requests without the header are unchanged.
  • Raw provider routes keep their existing header forwarding. Raw OpenAI chat still drops Idempotency-Key.

With the flag on:

  • Claiming: an atomic claim in D1 (migration 0023) precedes admission and dispatch.
    • The key (1–128 characters from A-Za-z0-9._:/+@-) is scoped to the authenticated principal, method and path.
    • A normalized digest binds it to the body, the requested model, the routing policy and the retention revision.
  • Outcomes:
    • A concurrent identical submission gets 409 request_in_progress with Retry-After: 1.
    • A changed request under the same key gets 422 idempotency_key_conflict.
    • Streaming keyed requests get 400 idempotency_stream_unsupported before any claim.
  • Replay: a completed response replays its stored status, body and bounded headers with X-MultiLLM-Idempotency: replayed. Replay calls no provider and reserves no usage.
  • Handoff: the provider handoff is recorded durably before submission.
    • A timeout, abandoned request, failed completion write or lost outcome blocks the key with 409 outcome_unknown, and never starts a second generation.
    • If the handoff acknowledgement itself fails, the reservation settles at zero, because no provider was contacted, and the key stays blocked; it does not become reusable.
  • Storage:
    • Response bytes go in R2 under managed-idempotency/, up to 1 MiB with at most 32 headers, and expire 24 hours after the claim.
    • D1 holds only hashed identity, state, expiry and an integrity-checked pointer.
    • Errors and partial results are never stored as successes.
    • Zero-content retention prevents storage and replay.
  • Fail-closed: missing storage returns 503 idempotency_store_unavailable before generation. There is no process-local fallback.

Details are in docs/managed-idempotency.md.

Worker

The private outbound handler mounts the managed-state dispatcher (/v1/managed-state/*). Paths are fixed, only POST from intelligence.internal is accepted, and unregistered domains return 404.

The idempotency domain is wired here. The reservations and tool-grants domains accept injected handlers, but nothing is injected; those features use their own routes.

Glue left for the Wave 4 integration PR

  • Register middleware.idempotency.register_idempotency through services/gateway_extensions.py, with the named idempotency_request_hook after authentication, retention and secret policy and before admission. Until it is registered, turning the flag on makes keyed managed requests fail closed with 503 idempotency_store_unavailable. Requests without a key are unaffected.
  • Pass the resolved policy revision for /v1/chat/completions and connect usage settlement.
  • Reservations keep their own private route, /v1/reservations (feat(budget): add opt-in durable ambiguous usage holds #44), and tool grants keep the model-state tool_grants operation (feat(mcp): add opt-in permission-aware deferred tool discovery #38). The managed-state reservations and toolGrants injection points stay unused, so there is only one route for each.
  • README line.

Stacking

This PR is based on feat/gateway-alerts (#46), so migrations 0018–0023 stay contiguous. Review only the top commit, feat(idempotency): add opt-in managed request idempotency; the merge commit brings in the lower layers.

Deployment

Apply the migration before merging to main. /ready checks the table, and a push to main does not apply D1 migrations:

npx wrangler d1 migrations apply multillm-intelligence --remote

Operators should also add an R2 lifecycle rule for the managed-idempotency/ prefix to remove expired and orphaned objects.

Shared registrations

  • Added tests/test_managed_idempotency_worker.mjs to test:worker.
  • Added MANAGED_IDEMPOTENCY_ENABLED to the Container env keys.
  • Added managed_idempotency to the D1 readiness tables.
  • Added worker/idempotency-d1.mjs and worker/managed-state-dispatch.mjs to the secret firewall inventory.
  • Added X-MultiLLM-Idempotency to the exposed CORS headers in Flask and the Worker.

Tests

  • New tests: tests/test_managed_idempotency.py and tests/test_managed_idempotency_worker.mjs. They cover:
    • the flag off;
    • missing keys;
    • claim, replay and conflicts;
    • in-progress and unknown outcomes;
    • handoff failure settling at zero;
    • raw passthrough unchanged;
    • the private dispatcher.
  • Full Python suite: 3,718 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,444 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 23 contiguous, additive migrations.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 4.

Accept Idempotency-Key on managed nonstreaming chat behind MANAGED_IDEMPOTENCY_ENABLED (default off). An atomic D1 claim (migration 0023) precedes admission and dispatch, binds the key to the principal, path, normalized body and routing policy, and records the provider handoff durably. Concurrent duplicates get 409 request_in_progress, a changed request under the same key gets 422, and a completed response replays from R2 with X-MultiLLM-Idempotency: replayed and no new provider call. With the flag off, the intelligence gateway keeps rejecting the header and raw provider routes are unchanged.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 24d66c0 Oct 09 2026, 12:42 AM

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6f07b6fc-eea4-483f-8044-ebf385a634d5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gitguardian

gitguardian Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37552414 Triggered Generic Password 9aea628 worker/alert-delivery.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@ALikesToCode
ALikesToCode merged commit b2d86a0 into feat/gateway-alerts Oct 9, 2026
3 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/managed-idempotency branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant